Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.40%. Comparing base (5aebee8) to head (dec1e4e).
⚠️ Report is 41 commits behind head on main.

@@            Coverage Diff             @@
##             main   #21541      +/-   ##
==========================================
- Coverage   68.45%   68.40%   -0.05%     
==========================================
  Files         428      429       +1     
  Lines       35383    35394      +11     
==========================================
- Hits        24221    24212       -9     
- Misses       9761     9772      +11     
- Partials     1401     1410       +9     

Run failed. @marcus-hodgson-antithesis said you need to change test type from Basic_test to Basic_k8s_test. However I don't see any place to switch that in https://github.com/antithesishq/antithesis-trigger-action

https://linuxfoundation.antithesis.com/report/AoQZ2hpIiK3E7UbJyV07QSlI/C2-sYK3t03a972_AYuITfbl-zs2alU4dUUDt468U2hM.html?auth=v2.public.eyJuYmYiOiIyMDI2LTAzLTI5VDEyOjQ1OjM5LjU1MTUzMTQwOVoiLCJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiQzItc1lLM3QwM2E5NzJfQVl1SVRmYmwtenMyYWxVNGRVVUR0NDY4VTJoTS5odG1sIiwicmVwb3J0X2lkIjoiQW9RWjJocElpSzNFN1ViSnlWMDdRU2xJIn19fae0UN_83uQ2YxT5gG9lkJu3fhuVGDrNgQeHjsVShC0MydVmH-wZ2cGpYLSi58uwkigwfAPF776TLr8hO82yowo

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nwnt, serathius

The full list of commands accepted by this bot can be found here.

The pull request process is described here

We have a passing run https://linuxfoundation.antithesis.com/report/gnMFNSkbTUOkpLxqq7y_vP-l/oQYD0X8p0pl3UkSCmRM-6b0L8McEnD0St5u351n-iKw.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0Ijoib1FZRDBYOHAwcGwzVWtTQ21STS02YjBMOE1jRW5EMFN0NXUzNTFuLWlLdy5odG1sIiwicmVwb3J0X2lkIjoiZ25NRk5Ta2JUVU9rcEx4cXE3eV92UC1sIn19LCJuYmYiOiIyMDI2LTAzLTI5VDIwOjA2OjUyLjM1NDAyMjE4NFoifbIaxbqyCV__bSDACYGii7iPrj-1nSJKH9zfFm3vd0YSkNYdHaRKQQYz_iRrGumWp3Kv_mfL6hm0r_MH8J8BjwY

It confirmed that antithesis.duration still accepts hours and not minutes.

To confirm with previous docker-compose runs, it didn't report following assertion errors:

• container: etcd1, exit code: 1 - that's good, we don't care about this one
• consistent_index isn't equal to snapshot index - could imply different failure injection and worse exploration?
• Linearization timeout - could imply less aggressive failure injection, but might show up if we increase test duration.

cc @marcus-hodgson-antithesis could you help review this migration, and also ask someone else from Antithesis that specializes in K8s to take a look too? Would be good to confirm that we will not regress our ability to reproduce issues.

@serathius the basic-k8s-test kubernetes endpoint only runs with network faults. The container faults / other faults that we currently have enabled on the docker setup, haven't been enabled yet on this kubernetes one.

I'm in the process of creating a custom endpoint for etcd where we can enable move over what we have from the current setup to this new one

The container faults / other faults that we currently have enabled on the docker setup, haven't been enabled yet on this kubernetes one.

Can we enable them and just migrate to K8s?

Yep, I've created a custom endpoint with the faults we had for docker @serathius

I've added a comment on the line of code we'd need to change to use it!

I've added a comment on the line of code we'd need to change to use it!

Don't see comment, might need to publish?

Yep you we're right. Forgot to publish the comment -_-

Forgot to publish the comment -_-

Yea, not the first time for me when GitHub was unnessesery casing friction.

Started run https://github.com/etcd-io/etcd/actions/runs/24122063300

/retest

@serathius: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Same configuration that worked for basic_k8s_test (basic_k8s_test run) doesn't work with etcd_k8s (etcd_k8s run)

Getting error:

7:01:57AM: ---- waiting on 2 changes [1/3 done] ----
7:02:00AM: ongoing: reconcile deployment/etcd-client (apps/v1) namespace: default
7:02:00AM:  ^ Waiting for 1 unavailable replicas
7:02:00AM:  L ok: waiting on replicaset/etcd-client-84b8d9b746 (apps/v1) namespace: default
7:02:00AM:  L ongoing: waiting on pod/etcd-client-84b8d9b746-7t5f8 (v1) namespace: default
7:02:00AM:     ^ Pending: ErrImageNeverPull, message: Container image "etcd-client:antithesis_dec1e4e969995813f82ffe8da80358c636b4992b" is not present with pull policy of Never
7:02:00AM: ongoing: reconcile statefulset/etcd (apps/v1) namespace: default
7:02:00AM:  ^ Waiting for 3 replicas to be ready
7:02:00AM:  L ongoing: waiting on pod/etcd-2 (v1) namespace: default
7:02:00AM:     ^ Pending: ErrImageNeverPull, message: Container image "etcd-server:antithesis_dec1e4e969995813f82ffe8da80358c636b4992b" is not present with pull policy of Never
7:02:00AM:  L ongoing: waiting on pod/etcd-1 (v1) namespace: default
7:02:00AM:     ^ Pending: ErrImageNeverPull, message: Container image "etcd-server:antithesis_dec1e4e969995813f82ffe8da80358c636b4992b" is not present with pull policy of Never
7:02:00AM:  L ongoing: waiting on pod/etcd-0 (v1) namespace: default
7:02:00AM:     ^ Pending: ErrImageNeverPull, message: Container image "etcd-server:antithesis_dec1e4e969995813f82ffe8da80358c636b4992b" is not present with pull policy of Never

Also possible that something broke between version 48 and 50.:

• Successful run 52aa7f48caa06b031ccb1af3a8e21713-48-5
• Failed run 8b346466af5340f038a00a847fafbce0-50-7

Talked with @marcus-hodgson-antithesis, I was wrong about previous run using k8s. Looks like only new run used K8s and everything looks correct even thou it didn't work. Action item is on Antithesis side to debug what went wrong.

Okay I've figured it out!

It looks like kapp is looking for an absolute image path. So, in Antithesis, we have the images:

us-central1-docker.pkg.dev/molten-verve-216720/linuxfoundation-repository/etcd-server:antithesis_dec1e4e969995813f82ffe8da80358c636b4992b

and

us-central1-docker.pkg.dev/molten-verve-216720/linuxfoundation-repository/etcd-client:antithesis_dec1e4e969995813f82ffe8da80358c636b4992b.

This makes sense because we are passing through those images in the antithesis.images parameter when kicking off a test (here).

@nwnt or @serathius is there anyway the manifests can use this absolute path when getting run in Antithesis (here)?

Yeah I think so. We can perhaps use something like kustomize to patch the manifest for running in the antithesis environment. What do you think @serathius ?